rustdoc-json: Postcard output by aDotInTheVoid · Pull Request #142642 · rust-lang/rust

aDotInTheVoid · 2025-06-17T20:46:32Z

What

rustdoc --output-format=postcard is like rustdoc-json, but using
https://postcard.rs/ / https://docs.rs/postcard/1.1.1/ instead of JSON.

Why

JSON Size and speed isn't great. People
want
more
speed,
and smaller docs. There are proposals to make the JSON smaller (and therefor
faster) by making field-names shorter, and omitting them when the value is the
default. But

How good is it?

In a very unscientific
benchmark for
aws-sdk-ec2, it's ~3.6x smaller (255MiB vs 69 MiB) and ~1.8x faster to
deserialize (1.6273 s vs 914.05 ms)

What's the metaformat

21 bytes of magic numbers
varint(u32) format version
Crate as usual

This way, users can look at the magic number to check it's a
rustdoc-json-postcard file, then read the version number to know if they can
decode it. Only then can they deserialize the Crate itself. I plan to write a
library that does this, so it's easy to do well.

What about the name

Right now, this means that the rustdoc::json module produces postcard & json
output. And the overall feature is still called "rustdoc json". There should
probably be a unified name for the both of these.

Maybe something like "Rustdoc IR Output"? But that can wait for a follow-up
commit I think.

Why is this a draft

Needs more tests of CLI flag interactions
Currently both HtmlRenderer and JsonRenderer are configures from the same options, we should change this
- #t-rustdoc > Using `RenderOptions` in less places.
- rustdoc: Don't pass RenderOptions to DocContext #147832
- The rest of this isn't needed for this.
~~I want to make it more principled how rustdoc before the format inspects .is_json() instead of the current hacks~~
- compiletest: pass rustdoc mode as param, rather than implicitly #148003
~~Compiletest changes should be spun into their own thing~~
- compiletest: pass rustdoc mode as param, rather than implicitly #148003
~~Docs~~

aDotInTheVoid · 2025-06-17T20:49:45Z

CC @obi1kenobi @nnethercote @LukeMathWalker

aDotInTheVoid · 2025-06-17T21:21:34Z

+pub mod postcard {
+
+    pub type Magic = [u8; 22];
+    pub const MAGIC: Magic = *b"\x00\xFFRustdocJsonPostcard\xFF";


A friend points out https://hackers.town/@zwol/114155807716413069, with advice on how to design a magic number.

Having 'Json' in there seems perverse :)

Sadly that link is now dead since they moved the server to masto.hackers.town and the posts no longer has the same ID, or it doesn't exists

It got archived: https://archive.is/0QDhX. Repeated here for posterity.

another day, another binary file format with a badly designed magic number

not gonna call it out specifically but here are some RFC2119 MUSTs for magic number design:

MUST be the very first N bytes in the file

MUST be at least four bytes long, eight is better

MUST include at least one byte with the high bit set

MUST include a byte sequence that is invalid UTF-8

SHOULD include a zero byte, but you can usually get away with having that be part of the overall version number that immediately follows the magic number (did I mention that you really SHOULD put an overall version number right after the magic number, unless you know and have documented exactly why it's not necessary, e.g. PNG?)

good examples: PNG, ELF

bad examples: GIF PE PDF

Here is a template. If you follow this template for your binary file format's magic number, you will be doing it better than a depressingly large number of senior software engineers.

First eight bytes of the file:

0xDC 0xDF X X x x (0x01 0x00 | 0x00 0x01)

0xDC 0xDF are bytes with the high bit set. Together with the next two bytes, they form a four-byte sequence that cannot appear in any valid ASCII, UTF-8, Corrected UTF-8, or UTF-16 (regardless of endianness) text document. This is not a perfectly bulletproof declaration that the file does not contain text, but it should be strong enough except maybe for formats like PDF that can't decide if they're structured text or binary.
X X x x: Four ASCII alphanumeric characters naming your file format. Make them clearly related to your recommended file name extension. I'm giving you four characters because we're running out of three-letter acronyms. If you don't need four characters, pad at the end with 0x1A (aka ^Z).

The first two of these (the uppercase Xes) must not have their high bits set, lest the "this is not text" declaration be weakened. For the other two (lowercase xes), use of ASCII alphanumerics is just a strong recommendation.
0x01 0x00 or 0x00 0x01: This is to be understood as a 16-bit unsigned integer in your choice of little- or big-endian order. It serves three functions. In descending order of importance:

It includes a zero byte, reinforcing the declaration that this is not a text file.

It demonstrates which byte ordering will be used throughout the file. It does not matter which order you choose, but you need to consciously choose either big- or little-endian and then use that byte order consistently throughout the file. Yes, I have seen cases where people didn't do that.

It's an escape hatch. If one day you discover that you need to alter the structure of the rest of the file in a totally incompatible way, and yet it is still meaningfully the same format, so you don't want to change the name characters, you can change the 0x01 to 0x02. We both hope that day will never come, but we both know it might.

aDotInTheVoid · 2025-06-19T20:11:43Z

Something @jamesmunns pointed out is that this means that reordering fields or enum variants in rustdoc-json-types will now require a FORMAT_VERSION bump. This could probably be detected in CI using postcard-schema.

More broadly, we should think about where (if at all) postcard-schema fits into this.

jamesmunns · 2025-06-19T20:27:11Z

As a note, I'm working on iterating on postcard and postcard-schema right now, postcard-schema-ng was just released, and is a form that might be releasable as a 1.0 soon, but I'd need to finish up the items at jamesmunns/postcard#241 to see if any additional iteration is required.

postcard-schema gets you two interesting pieces of data:

postcard-schema::Key can be used to generate an 8-byte hash of the schema and the string of your choice, as a const. This can be snapshotted to detect wire changes in CI.
postcard-schema::NamedType/postcard-schema-ng::DataModelType is the full reflection-style schema of the data type, which is also serializable as postcard data. This can be useful if you want the data to be archival: storing the schema inside the file itself, so you could still decode it even if the schema changes (using the postcard-dyn crate, giving you a serde_json::Value-like view of the data).

postcard is also getting a 2.0 soon, but it's important to note that the wire format is NOT changing. You will be able to use the library version v1.0 and v2.0 interchangably, wrt to serialization/deserialization (it's a breaking change because I'm removing some external crates that are now out of dates from my public API, it's likely your code won't need to change at all).

jamesmunns · 2025-06-19T20:28:54Z

A possibly useful form for the file format could be:

struct PostcardFile<T> {
    key: Key,
    schema: Option<Schema>,
    data: T,
}

I've considered "standardizing" this format a bit, maybe with a trailing CRC32.

aDotInTheVoid · 2025-06-19T22:52:06Z

postcard is also getting a 2.0 soon, but it's important to note that the wire format is NOT changing. [...]

Awesome! It'd be great to not have cobs and embedded-io in Cargo.lock (and that for all the users that care about performance).

A possibly useful form for the file format could be:

I think we definatly want to keep the magic number, so that consumers can tell if this file is rustdoc output at all, and a linear format version so they can tell if rustdoc is too old or too new for them if the schema's changed (vs a schema hash that only tells you that it's changed). Embedded the schema into the output itself is an interesting idea, I'll need to look more at it. But as long as both of these come after the magic number and linear format version, we should be fine to change them after the fact.

bors · 2025-06-29T11:15:41Z

☔ The latest upstream changes (presumably #143173) made this pull request unmergeable. Please resolve the merge conflicts.

…oundwork, r=jieyouxu compiletest: pass rustdoc mode as param, rather than implicitly Spun out of #142642 In the future, I want the rustdoc-json test suite to invoke rustdoc twice, once with `--output-format=json`, and once with the (not yet implemented) `--output-format=postcard` flag. Doing that requires being able to explicitly tell the `.document()` function which format to use, rather then implicitly using json in the rustdoc-json suite, and HTML in all others. r? `@jieyouxu` CC `@jalil-salame`

…oundwork, r=jieyouxu compiletest: pass rustdoc mode as param, rather than implicitly Spun out of rust-lang/rust#142642 In the future, I want the rustdoc-json test suite to invoke rustdoc twice, once with `--output-format=json`, and once with the (not yet implemented) `--output-format=postcard` flag. Doing that requires being able to explicitly tell the `.document()` function which format to use, rather then implicitly using json in the rustdoc-json suite, and HTML in all others. r? `@jieyouxu` CC `@jalil-salame`

…letest-groundwork, r=jieyouxu compiletest: pass rustdoc mode as param, rather than implicitly Spun out of rust-lang#142642 In the future, I want the rustdoc-json test suite to invoke rustdoc twice, once with `--output-format=json`, and once with the (not yet implemented) `--output-format=postcard` flag. Doing that requires being able to explicitly tell the `.document()` function which format to use, rather then implicitly using json in the rustdoc-json suite, and HTML in all others. r? `@jieyouxu` CC `@jalil-salame`

…oundwork, r=jieyouxu compiletest: pass rustdoc mode as param, rather than implicitly Spun out of rust-lang/rust#142642 In the future, I want the rustdoc-json test suite to invoke rustdoc twice, once with `--output-format=json`, and once with the (not yet implemented) `--output-format=postcard` flag. Doing that requires being able to explicitly tell the `.document()` function which format to use, rather then implicitly using json in the rustdoc-json suite, and HTML in all others. r? `@jieyouxu` CC `@jalil-salame`

epage · 2025-12-05T15:15:08Z

Cargo could potentially benefit from an alternative format (#t-cargo > `cargo metadata` performance @ 💬). Would be good for us to coordinate on the format used, at least when it comes to stabilizing.

…oundwork, r=jieyouxu compiletest: pass rustdoc mode as param, rather than implicitly Spun out of rust-lang/rust#142642 In the future, I want the rustdoc-json test suite to invoke rustdoc twice, once with `--output-format=json`, and once with the (not yet implemented) `--output-format=postcard` flag. Doing that requires being able to explicitly tell the `.document()` function which format to use, rather then implicitly using json in the rustdoc-json suite, and HTML in all others. r? `@jieyouxu` CC `@jalil-salame`

aDotInTheVoid · 2026-08-09T13:06:32Z

+                    IrOutputFormat::Postcard => "postcard",
+                };
+
+                dcx.fatal(format!(


This should have a rustdoc-ui test.

View changes since the review

aDotInTheVoid · 2026-08-09T13:07:19Z

    pub(crate) fn is_json_output(&self) -> bool {
-        self.output_format == OutputFormat::IrJson
+        // TODO: Rename this!
+        matches!(self.output_format, OutputFormat::Ir(_))


This needs renaming, but I'm not sure what to. Open to suggestions.

View changes since the review

aDotInTheVoid · 2026-08-09T13:18:09Z

+                        OutputFormat::Ir(IrOutputFormat::Json) | OutputFormat::CoverageJson => dcx.fatal(format!(
                            "the `--emit={typ}` flag is not supported with `--output-format=json`",
                        )),
+                        OutputFormat::Ir(IrOutputFormat::Postcard) => dcx.fatal(format!("the `--emit={typ}` flag is not supported with `--output-format=postcard`")),


This should have a rustdoc-ui test.

View changes since the review

rustbot · 2026-08-09T13:24:58Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Some changes occurred in src/tools/compiletest

cc @jieyouxu

rustdoc-json-types is a public (although nightly-only) API. If possible, consider changing src/librustdoc/json/conversions.rs; otherwise, make sure you bump the FORMAT_VERSION constant.

cc @CraftSpider, @Enselic, @obi1kenobi

aDotInTheVoid · 2026-08-09T13:25:49Z

Finally ready for review:

r? @obi1kenobi @GuillaumeGomez

CC @LukeMathWalker @jamesmunns @nnethercote

rustbot · 2026-08-09T13:25:52Z

Failed to set assignee to obi1kenobi: invalid assignee

Note: Only org members with at least the repository "read" role, users with write permissions, or people who have commented on the PR may be assigned.

obi1kenobi

I think it looks good! Nice work!

Is there a chance we might ever want to bump the postcard and JSON version numbers separately? For example, if we introduced a structural change in how the postcard format is represented without changing the Rust-side types. Or do we say that we'll just bump the JSON format number then too, even though we didn't really need to?

View changes since this review

obi1kenobi · 2026-08-09T14:59:26Z

+    /// Postcard allows reading these field from a file one at a time.
+    /// It's reccomended to check the magic constant matches before reading the format version


minor typo fixes:

Suggested change

/// Postcard allows reading these field from a file one at a time.

/// It's reccomended to check the magic constant matches before reading the format version

/// Postcard allows reading these fields from a file one at a time.

/// It's recommended to check the magic constant matches before reading the format version

## What `rustdoc --output-format=postcard` is like rustdoc-json, but using https://postcard.rs/ / https://docs.rs/postcard/1.1.1/ instead of JSON. ## Why JSON Size and speed isn't great. People [want](https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/rustdoc_json.20ideas/with/524453896) [more](https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/rustdoc-json.3A.20compressed.20output/with/462869885) [speed](https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/.28De.29serialization.20speed.20of.20JSON.20docs), and smaller docs. There are proposals to make the JSON smaller (and therefor faster) by making field-names shorter, and omitting them when the value is the default. But ## How good is it? In a [very unscientific benchmark](https://github.com/aDotInTheVoid/rustdocjson-encoding-bench) for aws-sdk-ec2, it's ~3.6x smaller (255MiB vs 69 MiB) and ~1.8x faster to deserialize (1.6273 s vs 914.05 ms) ## What's the metaformat - 21 bytes of magic numbers - varint(u32) format version - `Crate` as usual This way, users can look at the magic number to check it's a rustdoc-json-postcard file, then read the version number to know if they can decode it. Only then can they deserialize the `Crate` itself. I plan to write a library that does this, so it's easy to do well. ## What about the name Right now, this means that the `rustdoc::json` module produces postcard & json output. And the overall feature is still called "rustdoc json". There should probably be a unified name for the both of these. Maybe something like "Rustdoc IR Output"? But that can wait for a follow-up commit I think.

aDotInTheVoid · 2026-08-09T15:26:56Z

Is there a chance we might ever want to bump the postcard and JSON version numbers separately? For example, if we introduced a structural change in how the postcard format is represented without changing the Rust-side types. Or do we say that we'll just bump the JSON format number then too, even though we didn't really need to?

I think we should bump whenever either of them requires a bump. There are cases where only JSON is broken (field/varient rename), and where only postcard is broken (field/varient reorder). But I don't think the gains from seperate versioning (wider compatibilty ranges) are worth the costs (more complex implementation, harder to explain to contributors which format versions need to be bumped, harder to explain to tool writers).

aDotInTheVoid commented Jun 17, 2025

View reviewed changes

aDotInTheVoid mentioned this pull request Jun 19, 2025

Add CI check to ensure that rustdoc JSON FORMAT_VERSION is correctly updated #142677

Merged

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 29, 2025

aDotInTheVoid mentioned this pull request Oct 22, 2025

compiletest: pass rustdoc mode as param, rather than implicitly #148003

Merged

rust-cloud-vms Bot force-pushed the rdp branch 2 times, most recently from 82d6bfb to f0b5215 Compare August 9, 2026 13:05

This comment has been minimized.

Sign in to view

rust-cloud-vms Bot force-pushed the rdp branch from f0b5215 to d4b2074 Compare August 9, 2026 13:13

This comment has been minimized.

Sign in to view

aDotInTheVoid commented Aug 9, 2026

View reviewed changes

rust-cloud-vms Bot force-pushed the rdp branch from d4b2074 to c21ca39 Compare August 9, 2026 13:23

aDotInTheVoid marked this pull request as ready for review August 9, 2026 13:24

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 9, 2026

This comment has been minimized.

Sign in to view

rust-cloud-vms Bot force-pushed the rdp branch from c21ca39 to cc8ecc3 Compare August 9, 2026 14:59

obi1kenobi reviewed Aug 9, 2026

View reviewed changes

rust-cloud-vms Bot force-pushed the rdp branch from cc8ecc3 to be2d86f Compare August 9, 2026 15:18

rust-cloud-vms Bot force-pushed the rdp branch from be2d86f to 1bdaa4f Compare August 9, 2026 15:19

		/// Postcard allows reading these field from a file one at a time.
		/// It's reccomended to check the magic constant matches before reading the format version

Uh oh!

Uh oh!

Conversation

aDotInTheVoid commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How good is it?

What's the metaformat

What about the name

Why is this a draft

Uh oh!

aDotInTheVoid commented Jun 17, 2025

Uh oh!

aDotInTheVoid Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

nnethercote Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

jalil-salame Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

aDotInTheVoid Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

aDotInTheVoid commented Jun 19, 2025

Uh oh!

jamesmunns commented Jun 19, 2025

Uh oh!

jamesmunns commented Jun 19, 2025

Uh oh!

aDotInTheVoid commented Jun 19, 2025

Uh oh!

bors commented Jun 29, 2025

Uh oh!

epage commented Dec 5, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

aDotInTheVoid Aug 9, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aDotInTheVoid Aug 9, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aDotInTheVoid Aug 9, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustbot commented Aug 9, 2026

Uh oh!

aDotInTheVoid commented Aug 9, 2026

Uh oh!

rustbot commented Aug 9, 2026

Uh oh!

This comment has been minimized.

obi1kenobi left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

obi1kenobi Aug 9, 2026

Choose a reason for hiding this comment

Uh oh!

aDotInTheVoid Aug 9, 2026

Choose a reason for hiding this comment

Uh oh!

aDotInTheVoid commented Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

aDotInTheVoid commented Jun 17, 2025 •

edited

Loading

aDotInTheVoid Aug 9, 2026 •

edited by rustbot

Loading

aDotInTheVoid Aug 9, 2026 •

edited by rustbot

Loading

aDotInTheVoid Aug 9, 2026 •

edited by rustbot

Loading

obi1kenobi left a comment •

edited by rustbot

Loading